Compact eye imaging and eye tracking apparatus

ABSTRACT

An optical system for eye tracking is disclosed. The system includes a light guiding prism that guides light from an ocular object to an imaging system through multiple internal reflections. The light guiding prism may include one or more freeform surfaces having optical power.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. Provisional Patent ApplicationNo. 61/699,462, filed on Sep. 11, 2012 and U.S. Provisional PatentApplication No. 61/736,832, filed on Dec. 13, 2012, the disclosures ofwhich are incorporated herein by reference in their entirety.

FIELD OF THE INVENTION

The present invention relates generally to eye imaging and eye trackingdevices, and more particularly, but not exclusively, to a compacteye-glass-like wearable optical device that is capable of capturing aneye image and tracking eye gaze direction.

BACKGROUND OF THE INVENTION

Eye tracking is the process to measure the eye movement or the eye-gazedirection. Various eye tracking technologies have been developed for usein a variety of applications, such as vision research, human computerinterfaces, tele-surgery, product packaging, retail layout andadvertising research, visual communication, and various militaryapplications. One conventional method to track eye movement is to mounttwo cameras on a mount (e.g., a helmet), which capture images of eacheye directly. This approach results in a bulky design and an unsightlyappearance, which potentially may comprise research results.

In recent years, efforts have been made to design eye-glass likeeye-tracking devices that are light, portable, ergonomic, andaesthetically more pleasing than previous technologies. For example,Tobii Technology of Stockholm, Sweden has developed an eye trackingsystem (www.tobii.com) that mounts the eye imaging camera on the arm ofan eyeglass frame and captures an image of the eye through reflectionoff of the glass. The Tobii system has the advantage of an eyeglass likeform factor, and is light weight, however, in order to properly capturethe eye image through reflection off of glass, there must be sufficientclearance between the user's face and the glass surface to avoid theobstruction of the eye image by user's face or the imaging optics. Thisrequires that very wide eyeglass lenses be used to avoid interferencebetween the imaging path and any part of the user's face. The resultingoverall package does not resemble a conventional pair of eyeglasses, andas a result, test subjects are unlikely to use the Tobii product outsideof a laboratory environment.

Similarly, SensorMotoric Instruments GmbH or Teltow, Germany (“SMI”) hasdeveloped an eye tracking system (www.smivision.com), which also has aneyeglass-like appearance and is lightweight. In the SMI system, thecamera is mounted on the glass frame under the eye, and directly imagesthe eye. This results in a thick and bulky frame design, which must movethe camera far enough away from the face to avoid interference.Furthermore, in this system, the camera captures an eye image at a closedistance and from a slanted direction (i.e., at a high angle), whichresults the eye image suffering keystone distortion. This arrangementalso presents optical performance challenges due to the large depth offield necessary to accommodate all possible eye positions.

U.S. Pat. No. 6,735,328 discloses the use of a beamsplitter immersed inan eyeglass lens, off of which an eye is imaged by a vertically orientedcamera positioned above a user's line of sight. While the arrangement ofthis patent is an improvement over other designs discussed above, thefield of view of the camera in this arrangement is severely limited bythe disclosed geometry.

U.S. Pat. No. 6,943,754 discloses an eye tracking system including acamera. The camera images an eye using a bulky and unsightly visor thathangs in front of the face of a user.

U.S. Patent Application No. 2010/0220291 discloses providing an array ofoptical detection elements directly onto an eyeglass-like lens locatedin front of an eye. While this device appears to be somewhat similar toa pair of eyeglasses in form factor, it is necessarily expensive andcomplex to fabricate, as it requires the use of semi-conductorfabrication processes to fabricate optical detectors and theirassociated electronics on a curved glass lens substrate.

SUMMARY OF THE INVENTION

The present invention concerns an optical apparatus that is capable ofcapturing eye image and tracking eye gaze direction. The presentinvention aims to provide an eye imaging and gaze tracking device thatis compact and lightweight, has an eye-glass appearance and is wearablelike a pair of ordinary eyeglass. The present invention also offerssee-through capability which provides to the observer the capability ofviewing the surrounding environment through the glass with minimumdistortion and superior optical quality.

Embodiments of the invention are directed to using a thin prism betweenan eye and a camera system. The thin prism acts as a light-guide, whichfolds the imaging path between the camera and the eye. The thin prismalso optionally folds an illumination path between an illuminationsource and the eye, thereby providing on-axis illumination. Thisarrangement enables a thin, eyeglass like eye tracking device, whichcaptures a frontal (i.e., on-axis) or near frontal image of the eye at adistance while maintaining a light-weight, visually appealing formfactor.

Other embodiments of the invention use a corrective optical element toeliminate any deviation or aberrations in the see-though viewing pathintroduced by the thin prism, such that a user of the device cancomfortably see through the eye-tracker as normal. For example, in oneof it aspects, the invention may include a wedge prism having onlyplanar surfaces. This prism acts as a light guide to supply illuminationlight to the eye, as well as proving imaging light to the camera fromthe illuminated eye. In this embodiment, the invention includes acomplementary prism, which is arranged with respect to the thin prismsuch that the two prisms appear to the eye as a plane-parallel plate, orother weakly powered optic.

In alternative embodiments, an eye-tracker according to the inventionuses a free-form prism between the eye and a sensor. The freeform prismincludes one or more surfaces with optical power, which are used bothfor imaging of the eye onto the sensor, and for optical aberrationcontrol. In certain embodiments, the freeform prism is used inconjunction with, or exclusive of, additional focusing optics such as acamera outside of the prism.

In one embodiment, the invention is directed to an optical system foreye tracking. The system has a prism with a first, second and thirdsurface, and a first optical axis intersecting the first surface. Theprism also has a second optical axis that intersects, and isapproximately orthogonal to the third surface. The system also includesan imaging system and a sensor arranged along the second optical axis.The prism is configured such that an ocular object positioned about thefirst optical axis is imaged by the imaging system onto the sensorthrough the prism.

In another embodiment, the prism is configured such that light from anocular object positioned about the first optical axis enters the prismat the first surface and exits the prism through the third surface priorto being imaged by the imaging system onto the sensor.

In another embodiment, the prism is configured such that light from anocular object positioned about the first optical axis reflects off aninterior side of the second surface prior to exiting the prism throughthe third surface. In yet another embodiment, the prism is configuredsuch that light propagating along the first optical axis that enters theprism at the first surface and reflects at least twice within the prismto exit the third surface along the second optical axis.

In some embodiments, the prism is configured such that light from anocular object positioned about the first optical axis entering the prismalong the first axis reflects off an interior side of the second surfaceand reflects off an interior side of the first surface before exitingthe prism through the third surface. In other embodiments, the prism isconfigured such that the reflection off the interior side of the firstsurface occurs by total internal reflection.

In certain embodiments, the prism is configured such that light from anocular object positioned about the first optical axis entering the prismalong the first axis reflects off an interior side of the secondsurface, reflects off an interior side of the first surface, and againreflects off an interior side of the second surface before exiting theprism through the third surface.

In some embodiments, the first surface of the prism includes a highlyreflective coating for light within a predetermined wavelength rangeincident on the interior side of the first surface. In some embodiments,the reflective coating extends over only a portion of the first surface.The reflective coating, in some embodiments, comprises a dielectric thinfilm stack capable of high reflectivity for near infrared light.

In certain embodiments, the second surface of the prism has a coatinghaving high reflectivity for light within a predetermined wavelengthrange incident on the interior side of the second surface. In someembodiments, the predetermined wavelength range is between about 700 nmand 1.5 um. In certain embodiments, the high reflectivity coating is adielectric thin film stack having of high reflectivity for near infraredlight.

In certain embodiments, the imaging system of the system has a firstlens, a second lens, and a stop located between the first and secondlenses. In alternate embodiments, at least one of the first, second orthird surfaces is a freeform surface. In some embodiments, at least oneof the first, second or third surfaces has optical power.

Some embodiments include a corrective optical element arranged along thefirst optical axis, wherein the corrective optical element has a firstsurface adjacent to the second surface of the prism, and a secondsurface, and wherein the corrective optical element is arranged suchthat its second surface is approximately parallel to the first surfaceof the prism. Alternative embodiments of the system include a correctiveoptical element arranged along the first optical axis, wherein thecorrective optical element has a first surface adjacent to the secondsurface of the prism, and a second surface, and wherein the correctiveoptical element is arranged such the corrective optical elementcounteracts any visual distortion introduced by the prism when lookingthrough the prism and the corrective optical element along the firstoptical axis.

In some embodiments, the optical system has at least one light sourcearranged to illuminate an ocular object located about the first opticalaxis. In some embodiments, the light source comprises an infrared lightemitting diode. In certain embodiments, the light source is arranged todirect light along the second optical axis into the third surface, whilein others, the light source is arranged to directly illuminate an ocularobject without passing through the prism.

Other embodiments include a mount to position the prism, the imagingsystem and sensor a predetermined distance from an ocular object of asubject, and to align the ocular object to the first optical axis.

In other embodiments, the invention includes an eye tracking system withan optical sensor, an imaging system, and a prism. The prism has afirst, second and third surface. The prism also has a first optical axisapproximately orthogonal to the first surface, and a second optical axisapproximately orthogonal to the third surface. The prism is configuredto reflect light from an ocular object located about the first opticalaxis off of an interior side of the second surface through the thirdsurface, such the ocular object is imaged by the imaging system on theoptical sensor. The system also includes a programmable computerprocessor in electronic communication with the optical sensor, and anon-transitory computer readable medium in electronic communication withthe programmable computer processor, the non-transitory computerreadable medium having computer executable instructions encoded thereonto cause the programmable processor to recognize the position of anocular object located along the first optical axis.

In some embodiments, the prism is configured such that light enteringthe prism along the first axis reflects off an interior side of thesecond surface and reflects off an interior side of the first surfacebefore exiting the prism through the third surface. Other embodimentshave a corrective optical element arranged along the first optical axis,wherein the corrective optical element arranged such the correctiveoptical element counteracts any visual distortion introduced by theprism when looking through the prism and the corrective optical elementalong the first optical axis.

In some embodiments, the invention is directed to an optical system foreye tracking. The system includes an optical sensor, an imaging system,and a prism with a first, second and third surface. The prism isarranged such that light entering the prism from an ocular objectundergoes multiple reflections within the prism prior to exiting theprism to be focused by the imaging system. The system also has a lightsource for illuminating an ocular object to be imaged by the imagingsystem.

Embodiments of the invention have certain advantages, in particular,over the conventional systems referenced above. For example, by placinga dense optical medium (e.g., the material of the prism) in object spacebetween the camera and the eye, the system takes advantage of theoptical phenomenon of reduced distance by increasing the physicaldistance (along the folded optical path through the prism) between thecamera and the eye. This increased physical path length “pushes” thecamera a farther distance off of the optical axis of the eye as comparedto the location it would occupy if the path between the eye and thecamera were in air, and that path was folded, for example, with a simpleair-immersed fold mirror or beamsplitter. The consequence of this isthat the camera can be located well out of the normal clear aperture ofa conventional pair of eyeglass lenses, and does not therefore interferewith the subject's normal field of view. Yet at the same time, the eyeis imaged by the camera from an on-axis position.

Additionally, the use of a light guiding prism enables eye tracking overa greater field of view than with conventional devices. By using a lightguiding prism, a first reflective surface inclined at a relatively smallangle with respect to the eye's optical axis (e.g., approximately 27.5degrees, measured from the surface normal of the reflective surface) canbe used to fold light from a wide range of field positions. This allowsthe eye to be tracked over a wider range of positions as compared with,for example, the single fold arrangement described above with respect toU.S. Pat. No. 6,735,328.

Additionally, eye tracking devices according to embodiments of theinvention can be realized with very thin prisms by folding the opticalpath inside the prism, 5 mm or less, which is not unreasonably thick ascompared to a conventional pair of eyeglasses. While certain embodimentsuse total internal reflection to accomplish at least one reflectionwithin the light guiding prism, other embodiments use “hot mirrors” orreflective coatings that are reflective in the infrared while beingtransmissive in the visible spectrum to guide the imaging light up andaway from the eye's optical axis. The use of infrared reflectivecoatings eliminates the TIR requirement on certain reflections forachieving high reflective efficiency, thereby allowing the prism to usea small reflection angle. This results in the prism being thinner thanwould be the case if a TIR condition was required. Furthermore, thesecoatings do not interfere with the subject's ability to see real-worldscenes through the prism because imaging of the eye is done in theinfrared, and the coatings are invisible in the visible spectrum.

Additionally, certain embodiments of the invention use freeform, ratherthan planar prisms. Such prisms may have deterministic, aspheric,non-rotationally symmetric surfaces with optical power, which enablesthe prism to be used for focusing power and/or aberration control.

Additionally, by mating a guiding prism in the eye imaging path with acomplementary and corrective optical element, the subject's view ofreal-world scenes is unobstructed and aberration free.

These and other features, aspects, and advantages of the presentinvention will become better understood upon consideration of thefollowing detailed description and drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

Implementations of the invention will become more apparent from thedetailed description set forth below when taken in conjunction with thedrawings, in which like elements bear like reference numerals.

FIG. 1 is a schematic block diagram of an eye tracking system accordingto an embodiment of the invention.

FIG. 2 is a schematic illustration of an eye tracking system accordingto an embodiment of the invention using a planar prism and an optionalcorrective optical element.

FIG. 3 is a schematic illustration of an eye tracking system accordingto an embodiment of the invention using an inclined planar prism and anoptional corrective optical element.

FIG. 4 is a schematic illustration of an eye tracking system accordingto an embodiment of the invention using a freeform prism and an optionalcorrective optical element.

FIG. 5 is a schematic illustration of an eye tracking system accordingto an embodiment of the invention using a freeform prism having a planarfirst surface, a powered third surface, and an optional correctiveoptical element.

FIG. 6 is a schematic illustration of an eye tracking system accordingto an embodiment of the invention using a freeform prism supportingthree internal reflections and an optional corrective optical element.

FIG. 7 is a schematic illustration of a variety of illuminationarrangements according to embodiments of the invention.

FIG. 8 is a block diagram showing an exemplary image processing methodin accordance with an embodiment of the invention.

DETAILED DESCRIPTION OF THE INVENTION

The invention is described in preferred embodiments in the followingdescription with reference to the Figures, in which like numbersrepresent the same or similar elements. Reference throughout thisspecification to “one embodiment,” “an embodiment,” or similar languagemeans that a particular feature, structure, or characteristic describedin connection with the embodiment is included in at least one embodimentof the present invention. Thus, appearances of the phrases “in oneembodiment,” “in an embodiment,” and similar language throughout thisspecification may, but do not necessarily, all refer to the sameembodiment.

The described features, structures, or characteristics of the inventionmay be combined in any suitable manner in one or more embodiments. Inthe following description, numerous specific details are recited toprovide a thorough understanding of embodiments of the invention. Oneskilled in the relevant art will recognize, however, that the inventionmay be practiced without one or more of the specific details, or withother methods, components, materials, and so forth. In other instances,well-known structures, materials, or operations are not shown ordescribed in detail to avoid obscuring aspects of the invention.

FIG. 1 is a schematic block diagram of an eye tracking system accordingto an embodiment of the invention. In the system of FIG. 1, an imagingsystem 100 is provided, represented by a lens. Imaging system 100 imagesan ocular object located at eye 110 onto an optical sensor 105. Opticalsensor 105 can be any device that translates optical power into electricsignal, for example, a CMOS or CCD array. In accordance with theembodiment of FIG. 1, imaging system 100 images the ocular object ontosensor 105 through a prism 115. Prism 115 has at least two optical axes:a first optical axis 120, along which light from the ocular objectenters prism 115, and a second optical axis 125 along which light fromthe ocular object exits prism 115 before being focused onto sensor 105by imaging system 100.

The system of FIG. 1 also includes an illumination source 130, forexample, a light emitting diode (“LED”). The LED is located adjacent toor otherwise in close proximity to imaging system 100 such that itdelivers light along or parallel to the imaging optical axes 120, 125,but in a counter propagation direction. In other words, illuminationsource 130 sends illumination light down the same or a similar paththrough prism 115 to eye 110 that light from eye 110 takes on its way toimaging system 100.

The system of FIG. 1 also includes corrective optical element 135.Corrective optical element 135 compensates for optical aberrations,wedge, and other optical artifacts introduced by prism 115 when asubject looks through prism 115. In other words, corrective opticalelement 135 is configured to allow for light propagating to a subject'seye along a viewing path 140, to reach the eye without obscuring orotherwise distorting the view of a subject through both optical elements115, 135. For example, in the event that prism 115 presents a wedge to asubject along a viewing path 140, corrective optical element 135 isconfigured as a wedge oriented 180 degrees about viewing path 140 suchthat, together, elements 115 and 135 form a plane parallel plate, orsome other non-distorting optical element, along viewing path 140.

The system of FIG. 1 also includes a computer 145 in electroniccommunication with sensor 105 and non-illustrated non-transitorycomputer readable medium (e.g., a hard disk drive). Computer 145receives electronic information (i.e., image data) from sensor 105regarding the levels of optical power on the pixels of sensor 105, andapplies an eye-tracking algorithm to determine the location of an ocularobject within the field of view of the sensor 105.

FIG. 2 shows an embodiment of a system for eye tracking according toanother embodiment of the invention. In the embodiment of FIG. 2, animaging system 200 is provided comprising a plurality of lenses (i.e., afirst and second lens 202 a, b), having an aperture stop 203 locatedbetween the two. In one embodiment, the stop 203 is located at a secondsurface of first lens 202 a. The system of FIG. 2 also includes anoptical sensor 205, on which imaging system 200 forms an image of anocular object from an eye 210. Additional lenses may be included inimaging system 200 as desired, e.g., for aberration control.

The system of FIG. 2 includes optical sensor 205. Optical sensor 205 canbe any device that translates optical power into electric signal, forexample, a CMOS or CCD detector array. Together, imaging system 200 andsensor 205 may be provided (and referred to herein) as a camera.

The system of FIG. 2 includes a prism 215, which has three planarsurfaces, a first surface 216, a second surface 217 and a third surface218. Prism 215, imaging system 200 and sensor 205 are arranged andconfigured such that an ocular object located within a predeterminedfield of the view defined by the imaging system 200 and the sensor 205,is imaged through prism 215 onto sensor 205. The optical system has afirst optical axis 220, along which light from the eye 210 located atobjective plane 212 propagates as it enters prism 215 at its firstsurface 216. The first optical axis 220 is approximately orthogonal tothe first surface 216 of the prism 215. The optical system also has asecond optical axis 225, which is approximately orthogonal to the thirdsurface 218 of prism 215. The imaging system 200 and the sensor 205 arearranged along the second optical axis 225.

The optical system of FIG. 2 is optionally rigidly held in apre-determined position with respect to a subject's eye, i.e., with anon-illustrated mounting system. The mounting system may include ahead-mounted apparatus similar to a pair of conventional spectacles. Theimaging system 200 has a field of view defined by imaging system 200 andsensor 205. In one embodiment, the imaging system 200 has an effectivefocal length 7.5 mm and the sensor 205 has an active area of 5.7 mm×4.3mm, which provides a field of view of 41.6°×32° and covers an area of 20mm×15 mm at object plane 212 positioned at 18 mm from first surface 216.

In operation, an ocular object located in or on eye 210 is illuminatedwith a non-illustrated illumination source. Preferred illuminationsources include light emitting diodes (“LEDs”), emitting in the nearinfrared, and an integrated focusing lens, such are used in standard T3or T5 LED packages. As used herein, “near infrared” means light in thewavelength range of between 700 nm and 1.5 um. In some embodiments, thepreferred illumination wavelength is 850 nm. Exemplary arrangements forilluminating eye 210 are set forth in additional detail below withrespect to FIG. 7.

Illumination light reflects off an ocular object located in proximity toeye 210. In some embodiments, the eye pupil is the ocular object imaged.In other embodiments, the first Purkinje image, which is the virtualimage formed by the reflection of a point source off the anteriorsurface of the cornea is used. Any physical or optical object associatedwith the eye that can be uniquely identified and that will indicateocular position, pupil position, or gaze direction is an acceptableocular object that may be imaged and tracked within the scope of theinvention.

Upon illumination of the ocular object, light propagates along a firstoptical axis 220. First optical axis 220 is established by the designparameters of prism 215 and imaging system 200 as the optical axis ofimaging system 200 in object space before it is folded and translatedvia reflection and refraction through prism 215. A non-illustratedmounting system mounts the eye-tracker of FIG. 2 (or enforces theposition of a subject's head) such that eye 210 is placed on or aboutfirst optical axis 220 at or near objective plane 212. The field of viewof the eye tracker, which is a function of sensor 205 and imaging system200 is sufficient to image the ocular object of interest throughout avariety of field positions located about axis 220. In one embodiment,this field of view, or imaged eye area measures 20 mm (horizontal) by 15mm (vertical) for a sensor measuring 5.7 mm×4.3 mm.

Light from the illuminated ocular object enters prism 215 at prism 215'sfirst surface 216. In the embodiment of FIG. 2, first surface 216 isapproximately orthogonal to first optical axis 220, but this is not arequirement. After entering prism 215 and refracting at first surface216, light from the ocular object reflects at an interior side of prism215's second surface 217. Second surface 217 makes an angle with firstsurface 216, which in a preferred embodiments is 27.5 degrees. Secondsurface 217 has disposed thereon a highly reflective coating optimizedfor high reflection at the illumination wavelength. The highlyreflective coating on second surface 217 is also optimized to be highlyreflective at the relevant angles, i.e., at the incidence andreflectance angles for light received by the surface from the ocularobject and reflected from the surface, as well as for light receivedalong an illumination path, which encounters second surface 217 at thesame angles. In the embodiment of FIG. 2, the highly reflective coatingof surface 217 is capable of high reflectivity in the near infrared, andin particular, at the designated wavelength of 850 nm and over a rangeof angles between about 18° and 38°. The coating also has low or zeroreflectivity in visible spectrum (400 nm˜700 nm). In a preferredembodiment, highly reflective coating is a dielectric thin-film stackdesigned for use on an acrylic-air interface at the wavelengths and overthe angles referenced above. In other embodiments, the highly reflectivesurface is designed for immersion in acrylic. The design of highlyreflective surface may be performed according to conventional thin-filmdesign methods.

Upon reflection from the highly reflective second surface 217, lightfrom the ocular structure propagates back through prism 215 to reflectoff of an interior side of first surface 216. In the embodiment of FIG.2, the angle between first and second surfaces 216, 217 is chosen suchthat all rays in the imaging ray bundle for all field points intersectan interior side of the first surface 216 above the critical angle, suchthat the illustrated reflection off of first surface 216 occurs by totalinternal reflection (“TIR”). This condition, however, is not arequirement. In other embodiments, a region of first surface 216 (e.g.,a region of the surface closer to imaging system 200) includes a highlyreflective coating similar to the coating on second surface 217. As canbe seen from FIG. 2, rays intersecting an interior side of first surface216 in the region of that surface closer to axis 220 do so at largeangles (measured with respect to the surface normal). As one travelsvertically up surface 216 in the direction of camera 200, it is apparentthat the reflections occur at increasingly small angles. If, at somepoint, these incident angles become too small to support TIR, theportion of first surface 216 over which the rays would refract out ofprism 215 can be coated with an infrared highly reflective coating.

Upon reflection from first surface, light from the ocular structurepropagates still through prism 215 along a second optical axis 225 untilencountering third surface 218, whereupon, the imaging light exits prism215 and refraction occurs. At this point, the light is collected byimaging system 200 and focused onto sensor 205. Second optical axis 225is the optical axis of imaging system 200, or is the axis of rotationalsymmetry for lenses 202 a, b.

As can be seen from FIG. 2, in the event that prism 215 is fabricatedusing a traditional cut-and-polish process (e.g., in the event that itis fabricated in glass), superfluous regions 245 may be created. Thesemay occur in the corners, in particular. In certain embodiments, weightis saved by cutting off these unnecessary regions with corner cuts. Inthe event that prism 215 is molded or cast in a polymer material, prism215 may be fabricated using the minimum possible material, without theinclusion of superfluous regions 245.

In a preferred embodiment, the eye tracking system of FIG. 2 is mountedto a subject's head, for example, in a form-factor similar in size,weight and appearance to conventional eyeglasses. Such a form factor isuseful when conducting real-world testing and data-gathering, sinceoftentimes, the goal of eye tracking in such circumstances is todetermine how a subject's eyes interact with real-world objects (e.g.,other automobiles, products on a store shelf, etc.) In such cases, then,it is desirable to provide the subject with as unobstructed a view ofthe real-world scene as possible, while still monitoring the ocularobject being observed. Thus, it is desirable to preserve an unobstructedand undistorted view path 240 from a real world object space to asubject's eye. In certain optional embodiments, this is accomplished byproviding a corrective optical element 235. Corrective optical element235 is arranged and configured to eliminate any wedge or other opticalaberration introduced by looking through prism 215 along viewing path240. In the case of FIG. 2, where prism 215 is a three sided (in crosssection) planar prism, this is accomplished by providing a complementaryprism as element 235 in close proximity to prism 215. In particular,element 235 includes a first surface 236 and a second surface 237. Firstsurface 236 is arranged proximate to second surface 217 of first prism215, and second surface 237 is arranged to be parallel to first surface216 of prism 215, such that when the subject looks through prism 215 andcorrective element 235, the user is effectively looking through a planeparallel plate. In a preferred embodiment corrective element 235 iscemented to, or is otherwise in optical contact with prism 215. In oneembodiment, both first prism 215 and corrective element 235 are made ofacrylic and the cemented resulting thin plate has a thickness of 5.5 mm.

As is set forth above, prism 215 in FIG. 2 is arranged and configuredsuch that its first surface 216 is orthogonal to first optical axis 220.However, this is not a requirement. FIG. 3 shows an eye tracking systemaccording to an embodiment of the invention in which a first surface 316is inclined at an angle with respect to first optical axis 320.

As in the system of FIG. 2, the system of FIG. 3 includes an imagingsystem 300, including lenses 302 a, b, and stop 303. As in the system ofFIG. 2, additional lenses or other optical elements may be added asnecessary for aberration control or other purposes. The system of FIG. 3also includes sensor 305. Imaging system 300 images an ocular objectlocated at or near eye 310 onto sensor 305.

As in the system of FIG. 3, as light from eye 310 is imaged, it travelsthrough prism 315, where it is folded twice such that a first ororiginal optical axis 320 is folded and translated to a second opticalaxis 325 along which sensor 305 and the elements of imaging system 300are arranged. Prism 315 includes a first, second and third surfaces 316,317, 318. Surface 317 includes an infrared high reflectivity coating,such that infrared light entering first surface 316 is reflected towardsecond surface 316. At internal side of the surface 316, the lightreflects, either through TIR, or through a selectively applied infraredhigh reflectivity coating covering only the upper part of surface 316such that the coating does not occlude the original light path into theprism through the first surface 316. After reflection from surface 317,light exits the prism 315 at third surface 318, where it is imaged byimaging system 300 onto sensor 305.

As in the system of FIG. 2, superfluous regions 345 of prism 315 may betrimmed or omitted. Additionally, to compensate for the wedge presentedto the subject when looking through prism 315, the system of FIG. 3 mayinclude an optional corrective optical element 335, which together withprism 315, forms a plane parallel plate with respect to the subject'seye 310 along a see-through direction 340.

FIG. 4 shows a compact eye tracker using a freeform prism. Unlike thesystems of FIGS. 2 and 3, which use planar surfaces, the surfaces ofprism 415 (surfaces 416, 417 and 418) are so-called “freeform” surfaces,in that their surface figure can be arbitrarily defined by design. Forexample, surfaces 416, 417 and 418 can be an aspherical, an anamorphicaspherical, an xyp polynomial aspherical, or other types of freeformsurfaces. The use of freeform surfaces enables the reduction of thenumber of lenses in imaging system 400, which in the case of theembodiment of FIG. 4, is a single lens 402, stop 403, and poweredfreeform surfaces 416, 417, 418. Additional aberration control isaccomplished in the embodiment of FIG. 4 by restricting the field overwhich the imaging system 400 must perform. In addition to providingfocusing power and aberration control, surfaces 416, 417 and 418 operateas corresponding surfaces described above with respect to prisms 215 and315. Light enters surface 416 from eye 410 and reflects from surface417, which carries a high reflectivity coating optimized for the nearinfrared. Light then reflects from surface 416 through total internalreflection. The system of FIG. 4 also optionally includes correctiveoptical element 435 to provide power, wedge and aberration free viewingalong view path 440.

In the system of FIG. 4, as in certain other embodiments when a freeformprism's first surface 416 is non-planar, corrective optical element 435may be designed such that any ray traced from the pupil of eye 410,which will refract upon entering prism 415 at first surface 416, hasthat refraction effectively cancelled upon exiting a second surface ofcorrective element 435. In other words, the surface figure of element435 is designed such that there is no net change in angle from raystraced from pupil to the real world object space beyond the system,although such rays may be axially displaced because of the presence ofthe dense optical medium of elements 415 and 435 between the eye andreal world object space. Such a design has the effect of creating no netchange in angle of rays propagating to eye 410 along real world viewingpath 440.

Referring now to FIG. 5, a light guiding prism 515 may be provided inwhich all surfaces but the third surface 518 is planar. Third surface518 is a freeform surface i.e., an aspherical, an anamorphic aspherical,an xyp polynomial aspherical, or some other type of freeform surface. Asin the system of FIG. 4, the use of freeform surface 518 enables areduction in the number of elements in imaging system 500, which in thecase of the system of FIG. 5, includes lens 502, stop 503 and freeformrefractive surface 518. Lens 502 may be an asphere to provide additionalaberration correction as well as focusing power. As in the systemsdescribed above, the prism 515 of the system of FIG. 5 has a secondsurface 517 with a high reflectivity IR coating, and a first surface516, which directs light from eye 510, through prism 515 to imagingsystem 505 by TIR, or a selectively applied infrared high reflectivitycoating. The system of FIG. 5 also includes an optional correctiveoptical element 535, which provides a distortion free viewing path 540for the subject.

In this embodiment, the optical axis 520 is inclined at an angle withrespect to object plane 512 of the eye 510, as compared with the systemof FIG. 2 where optical axis 220 is orthogonal to the object plane 212.The tilt of the optical axis 520 introduces a modest amount of keystonedistortion into the system of FIG. 5, which can be compensated in anumber of ways, for example, by tilting sensor 505, or by designingimaging system 500 with sufficiently large depth of focus/depth offield.

Referring now to FIG. 6, there is shown an eye tracking system using afreeform prism 615 in which all of the refractive and reflectivesurfaces 616, 617, 618 are freeform surfaces, which could be anaspherical, an anamorphic aspherical, an xyp polynomial aspherical, orother types of freeform surfaces. As in previously describedembodiments, the prism performs as a light guiding device as well as animage forming device. In the embodiment of FIG. 6, rays originating inan object plane 612 (e.g., at eye 610) centered on a first optical axispass through prism 615's first surface (616), and are then reflectedthree times inside the prism by surfaces 617, 616, and again by surface617. The first reflection from surface 617 is assisted with a highlyreflective coating on that surface; however, this coating need not becontinuous as reflections from the region of surface 617 nearest to theimaging system 600 will occur by TIR. The intermediate reflection fromthe first surface occurs through TIR. After multiple reflections, lightfrom the eye 610 exits the prism at refractive surface 618. Imagingsystem 600 then collects the rays and forms an image at sensor 605. Inthe system of FIG. 6, the imaging system is realized by all of prism615's freeform surfaces plus aspheric lens 602 and stop 603. As inpreviously described embodiments, in order to enable the see-throughcapability, a freeform corrective optical element 635 is cemented withthe freeform prism 615 to correct the viewing axis deviation andundesirable aberrations introduced by the prism to the real world scene.The freeform prism itself is trimmed to a thin profile to reduce weightas shown in embodiment described above.

Referring now to FIG. 7, there is shown a variety of illuminationarrangements usable in conjunction with the various embodiments of theinvention described above. FIG. 7 a shows an off-axis illuminationarrangement, whereby a plurality of lensed infrared LEDs 707 arearranged around a prism assembly, which itself includes a prism 715 anda corrective optical element 735. LEDs 707 provide off axis illumination708 to eye 710, which results in the formation of a darkened pupilimage.

FIG. 7 b shows an on-axis illumination arrangement. In the arrangementof FIG. 7 b, one or more light sources 707 is placed near the imagingsystem components such that light is directed along the same opticalpath as imaging light, but in the opposite direction. In a preferredembodiment, light sources 707 are located near the aperture stop used inthe imaging system, and the beams 708 generated by light sources 707 aredirected along the second optical axis described above with respect toFIG. 1. In the arrangement of FIG. 7 b, illumination light enters prism715 through third surface 718, and reflects off first surface 716, forexample, by TIR. Illumination light then reflects off second surface717, exits prism 715 at first surface 716 along the first optical axisin the direction of eye 710. On axis-illumination supplied in thismanner creates an image of a bright pupil. As in the embodimentsdescribed above, second surface 717 may include a high reflectivity IRcoating.

FIG. 8 shows an exemplary image processing process in accordance with anembodiment of the invention. According to the method of FIG. 8, animaging sensor receives an image of an ocular object. The sensortranslates the optical image into an electrical signal generating imagedata, which is received by a computer having a programmable processor incommunication with electronic storage (i.e., a non-transitory computerreadable medium having computer executable instructions). The computerapplies one or more feature detection algorithms to the image datareceived from the sensor, to detect ocular features. The computer thencalculates the gaze direction of the subject's eye on the basis of thedetected ocular feature. The computer then provides the calculated gazedirection to an application.

While the preferred embodiments of the present invention have beenillustrated in detail, it should be apparent that modifications andadaptations to those embodiments may occur to one skilled in the artwithout departing from the scope of the present invention as set forthin the following claims.

What is claimed is:
 1. An optical system for eye tracking, comprising: aprism having a first, second and third surface, the prism having a firstoptical axis intersecting the first surface, and a second optical axisintersecting and approximately orthogonal to the third surface; animaging system and a sensor arranged along the second optical axis,wherein, the prism is configured such that an ocular object positionedabout the first optical axis is imaged by the imaging system onto thesensor through the prism.
 2. The optical system of claim 1, wherein theprism is configured such that light from an ocular object positionedabout the first optical axis enters the prism at the first surface andexits the prism through the third surface prior to being imaged by theimaging system onto the sensor.
 3. The optical system of claim 2,wherein the prism is configured such that light from an ocular objectpositioned about the first optical axis reflects off an interior side ofthe second surface prior to exiting the prism through the third surface.4. The optical system of claim 2, wherein the prism is configured suchthat light propagating along the first optical axis that enters theprism at the first surface reflects at least twice within the prism toexit the third surface along the second optical axis.
 5. The opticalsystem of claim 2, wherein the prism is configured such that light froman ocular object positioned about the first optical axis entering theprism along the first axis reflects off an interior side of the secondsurface and reflects off an interior side of the first surface beforeexiting the prism through the third surface.
 6. The optical system ofclaim 5, wherein the prism is configured such that the reflection offthe interior side of the first surface occurs by total internalreflection.
 7. The optical system of claim 5, wherein the first surfaceincludes a highly reflective coating for light within a predeterminedwavelength range incident on the interior side of the first surface. 8.The optical system of claim 7, wherein the reflective coating extendsover only a portion of the first surface.
 9. The optical system of claim7, wherein the highly reflective coating comprises a dielectric thinfilm stack having of high reflectivity for near infrared light.7. 10.The optical system of claim 2, wherein the prism is configured such thatlight from an ocular object positioned about the first optical axisentering the prism along the first axis reflects off an interior side ofthe second surface, reflects off an interior side of the first surface,and again reflects off an interior side of the second surface beforeexiting the prism through the third surface.
 11. The optical system ofclaim 3, wherein the second surface includes a coating having highreflectivity for light within a predetermined wavelength range incidenton the interior side of the second surface.
 12. The optical system ofclaim 11, wherein the predetermined wavelength range is between about700 nm and 1.5 um.
 13. The optical system of claim 11, wherein thecoating comprises a dielectric thin film stack having of highreflectivity for near infrared light.
 14. The optical system of claim 1,wherein the imaging system comprises a first lens, a second lens, and astop located between the first and second lenses.
 15. The optical systemof claim 1, wherein at least one of the first, second or third surfacesis a freeform surface.
 16. The optical system of claim 1, wherein atleast one of the first, second or third surfaces has optical power. 17.The optical system of claim 1, further comprising a corrective opticalelement arranged along the first optical axis, wherein the correctiveoptical element has a first surface adjacent to the second surface ofthe prism, and a second surface, and wherein the corrective opticalelement is arranged such that its second surface is approximatelyparallel to the first surface of the prism.
 18. The optical system ofclaim 1, further comprising a corrective optical element arranged alongthe first optical axis, wherein the corrective optical element has afirst surface adjacent to the second surface of the prism, and a secondsurface, and wherein the corrective optical element is arranged such thecorrective optical element counteracts any visual distortion introducedby the prism when looking through the prism and the corrective opticalelement along the first optical axis.
 19. The optical system of claim 1,further comprising a light source arranged to illuminate an ocularobject located about the first optical axis.
 20. The optical system ofclaim 19, wherein the light source comprises an infrared light emittingdiode.
 21. The optical system of claim 19, wherein the light source isarranged to direct light along the second optical axis into the thirdsurface.
 22. The optical system of claim 19, wherein the light source isarranged to directly illuminate an ocular object without passing throughthe prism.
 23. The optical system of claim 1, further comprising a mountto position the prism, the imaging system and sensor a predetermineddistance from an ocular object of a subject, and to align the ocularobject to the first optical axis.
 24. An eye tracking system,comprising: an optical sensor; an imaging system; a prism having afirst, second and third surface, the prism having a first optical axisapproximately orthogonal to the first surface, and a second optical axisapproximately orthogonal to the third surface, the prism configured toreflect light from an ocular object located about the first optical axisoff of an interior side of the second surface through the third surface,such the ocular object is imaged by the imaging system on the opticalsensor; a programmable computer processor in electronic communicationwith the optical sensor, and a non-transitory computer readable mediumin electronic communication with the programmable computer processor,the non-transitory computer readable medium having computer executableinstructions encoded thereon to cause the programmable processor torecognize the position of an ocular object located along the firstoptical axis.
 25. The system of claim 24, wherein the prism isconfigured such that light entering the prism along the first axisreflects off an interior side of the second surface and reflects off aninterior side of the first surface before exiting the prism through thethird surface.
 26. The system of claim 25, further comprising acorrective optical element arranged along the first optical axis,wherein the corrective optical element arranged such the correctiveoptical element counteracts any visual distortion introduced by theprism when looking through the prism and the corrective optical elementalong the first optical axis.
 27. An optical system for eye tracking,comprising: an optical sensor; an imaging system; a prism having afirst, second and third surface, wherein the prism is arranged such thatlight entering the prism from an ocular object undergoes multiplereflections within the prism prior to exiting the prism to be focused bythe imaging system.
 28. The optical system of claim 27, furthercomprising a light source for illuminating an ocular object to be imagedby the imaging system.